{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "In this example you will learn, how to use extract structured data when parsing ACORD forms using Tensorlake DocumentAI. To learn more about ACORD form processing [check out the Tensorlake docs](https://docs.tensorlake.ai/use-cases/insurance-financial-services/acord-form-processing)"
      ],
      "metadata": {
        "id": "ilY98cmV6Nk9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorlake"
      ],
      "metadata": {
        "collapsed": true,
        "id": "mligrnYVZhmk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4Un5rSYjZcIp"
      },
      "outputs": [],
      "source": [
        "# Import libraries\n",
        "from tensorlake.documentai import DocumentAI\n",
        "from tensorlake.documentai.models import (\n",
        "    EnrichmentOptions,\n",
        "    PageClassConfig,\n",
        "    ParsingOptions,\n",
        "    StructuredExtractionOptions,\n",
        "    ParseStatus\n",
        ")\n",
        "from tensorlake.documentai.models.enums import (\n",
        "    ChunkingStrategy,\n",
        "    TableOutputMode,\n",
        "    TableParsingFormat,\n",
        ")\n",
        "import time\n",
        "import json"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%env TENSORLAKE_API_KEY=your_api_key"
      ],
      "metadata": {
        "id": "Z0CJaPdHZc0P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Specify Structured Data Extraction\n",
        "\n",
        "Create a simple JSON schema to specify what structured data you want extracted from the document"
      ],
      "metadata": {
        "id": "NmCSH2CstazZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# JSON schema to extract relevant data in a structured format\n",
        "structured_schema = {\n",
        "  \"title\": \"Acord125ApplicantInfo\",\n",
        "  \"type\": \"object\",\n",
        "  \"properties\": {\n",
        "    \"agencyName\": { \"type\": \"string\" },\n",
        "    \"agencyContactName\": { \"type\": \"string\" },\n",
        "    \"agencyPhone\": { \"type\": \"string\" },\n",
        "    \"applicantName\": { \"type\": \"string\" },\n",
        "    \"applicantAddress\": { \"type\": \"string\" },\n",
        "    \"businessPhone\": { \"type\": \"string\" },\n",
        "    \"fein\": { \"type\": \"string\" },\n",
        "    \"policyNumber\": { \"type\": \"string\" },\n",
        "    \"effectiveDate\": { \"type\": \"string\" },\n",
        "    \"expirationDate\": { \"type\": \"string\" },\n",
        "    \"typeOfOrganization\": { \"type\": \"string\" },\n",
        "    \"linesOfBusiness\": {\n",
        "      \"type\": \"array\",\n",
        "      \"items\": { \"type\": \"string\" }\n",
        "    }\n",
        "  }\n",
        "}"
      ],
      "metadata": {
        "id": "T8sACwLOZc47"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parse the Document\n",
        "To use the Tensorlake Python SDK, you need to:\n",
        "1. Create a Tensorlake Client\n",
        "2. Specify a file, in this case we have a sample ACORD form\n",
        "3. Specify Parsing Options, including page numbers and a chunking strategy for parsing\n",
        "4. Specify Structured Extraction Options, which need to include a schema name and schema at minimum.\n",
        "5. Initiate the parsing job and wait until it compeltes successfully"
      ],
      "metadata": {
        "id": "Y3LyACJEtlSS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a Tensorlake Client, this will reference the `TENSORLAKE_API_KEY` environment variable you set above\n",
        "doc_ai = DocumentAI()\n",
        "\n",
        "# Reference to the 29-page ACORD Form (125, 823, and 140)\n",
        "file_id = \"https://pub-226479de18b2493f96b64c6674705dd8.r2.dev/acord_form_125_823_140.pdf\"\n",
        "\n",
        "# Configure parsing with structured schema\n",
        "parsing_options = ParsingOptions(\n",
        "    page_range = \"1-4\", # Specify the page range you want the structured data to reference (e.g. only the 125 Form pages)\n",
        "    chunking_strategy=ChunkingStrategy.PAGE\n",
        ")\n",
        "\n",
        "structured_extraction_options = StructuredExtractionOptions(\n",
        "    schema_name=\"ACORD Form\",\n",
        "    json_schema=structured_schema\n",
        ")\n",
        "\n",
        "# Parse the document with the specified extraction options for structured data\n",
        "parse_id = doc_ai.parse(file_id, parsing_options=parsing_options, structured_extraction_options=[structured_extraction_options])\n",
        "\n",
        "# Wait for completion\n",
        "result = doc_ai.get_parsed_result(parse_id)\n",
        "print(f\"Parse job {parse_id} is {result.status}, waiting...\")\n",
        "while result.status in [ParseStatus.PENDING, ParseStatus.PROCESSING]:\n",
        "    time.sleep(5)\n",
        "    result = doc_ai.get_parsed_result(parse_id)\n",
        "    if result.status == ParseStatus.SUCCESSFUL:\n",
        "        print(f\"Parse job {parse_id} is {result.status}\")\n",
        "        break\n",
        "    print(f\"Parse job {parse_id} is {result.status}, waiting...\")"
      ],
      "metadata": {
        "id": "vjVbPNDiZc2l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tensorlake Parsing Output\n",
        "\n",
        "In one single DocumentAI API call, Tensorlake returns both the full markdown content of the document and the structured data in JSON format."
      ],
      "metadata": {
        "id": "bxPKR73NgOMc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Review the Markdown Chunks Output"
      ],
      "metadata": {
        "id": "1RAKCh3ftMAq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the markdown from extracted data\n",
        "for index, chunk in enumerate(result.chunks):\n",
        "    print(f\"Chunk {index}:\")\n",
        "    print(chunk.content)\n",
        "    if index == 3:\n",
        "      break"
      ],
      "metadata": {
        "id": "ikeITyB8eBVv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Review the Structured Data Output"
      ],
      "metadata": {
        "id": "b5-D92xQvDIX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Print the structured data output\n",
        "result.structured_data"
      ],
      "metadata": {
        "id": "Ez3xSJ6beAvD"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}